Conversational Agents on Your Behalf

Conversational Agents on Your Behalf: Opportunities and Challenges of Shared Autonomy in Voice Communication for Multitasking

The collaboration project with the leading author, a Sony CSL Summer Intern, Yi Fei Cheng (Carnegie Mellon University), and Hirokazu Shirado (Carnegie Mellon University)

Abstract

Advancements in computational agents will enable them to act as surrogates for users in online communication, promising enhanced productivity by supporting multitasking. This capability may be especially powerful when combined with human control, allowing users to retain agency while achieving better performance than either human or agent alone. However, it remains unclear how people might leverage this technology to multitask effectively. We present a study with 18 dyads exploring how users employ automated responses to support an arithmetic task while staying engaged in a voice call. Participants multitasked with a conversational agent under three levels of autonomy: none, shared, and full. Our findings indicate that fully automated systems can maintain conversational engagement, enabling users to multitask effectively. Surprisingly, shared autonomy hindered this ability. Based on our results, we discuss implications for designing shared autonomy in conversations, highlighting new considerations and challenges.

https://dl.acm.org/doi/10.1145/3706598.3714017

Yi Fei Cheng, Hirokazu Shirado, and Shunichi Kasahara. 2025. Conversational Agents on Your Behalf: Opportunities and Challenges of Shared Autonomy in Voice Communication for Multitasking. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 160, 1–18. https://doi.org/10.1145/3706598.3714017